A Preprocessing Framework and Approach for Web Applications

In the process of abstract extraction, two heuristics are used. First, the content of a page is organized semantically by paragraphs. Second, readers can quickly catch the ideas of the page with only a small quantity of sentences, and the sentences are generally located by key-words. Based on these heuristics, we can acquire an algorithm that will simulate the browsing manner of a reader to extract the abstract using the keywords obtained in the previous phase. The algorithm first divides the topic content into several paragraphs according to the tag tree, and then the sentences with high weights are picked from each paragraph to make up the abstract of the page. Following, two important subprocesses are illustrated in detail. 1. Identify the paragraphs of the content Fortunately, the structure of container tags describes the layout of HTML pages, which makes it possible to identify the paragraphs of the content. In the topic content tree, first locate the node that is the lowest common ancestor of all leaf nodes (we call it topic root). Topic root corresponds with the tag that exactly embeds all topic content nodes. Then, the son nodes of the topic root correspond with the paragraphs of the topic content. Figure 3 shows the process of identifying paragraphs. In (3) of Figure 3, each p block is a paragraph.